Generating Optimal Policies for High-level Plans with Conditional Branches and Loops

We are concerned with generating optimal policies for Markov decision processes that are represented as high-level plans with conditional branches and loops. Often complex planning processes can be broken down into elementary plan steps with associated restricted sets of actions. These plan steps can be combined to form high-level plans using a simple programming language specifying conditionals, loops, and sequences involving the plan steps as primitive statements. It is infeasible to directly generate and solve the underlying Markov decision process, since the size of the state space is exponential in the size of a high-level plan. We address the problem of eeciently computing an optimal policy by taking advantage of locality structure in the high-level plan. The main result is the speciication and analysis of an algorithm that takes as input a high-level plan and provides as output an optimal policy for the underlying Markov decision process.